Prediction models for early diagnosis of actinomycotic osteomyelitis of the jaw using machine learning techniques: a preliminary study

Background This study aimed to develop and validate five machine learning models designed to predict actinomycotic osteomyelitis of the jaw. Furthermore, this study determined the relative importance of the predictive variables for actinomycotic osteomyelitis of the jaw, which are crucial for clinical decision-making. Methods A total of 222 patients with osteomyelitis of the jaw were analyzed, and Actinomyces were identified in 70 cases (31.5%). Logistic regression, random forest, support vector machine, artificial neural network, and extreme gradient boosting machine learning methods were used to train the models. The models were subsequently validated using testing datasets. These models were compared with each other and also with single predictors, such as age, using area under the receiver operating characteristic (ROC) curve (AUC). Results The AUC of the machine learning models ranged from 0.81 to 0.88. The performance of the machine learning models, such as random forest, support vector machine and extreme gradient boosting was significantly superior to that of single predictors. Presumed causes, antiresorptive agents, age, malignancy, hypertension, and rheumatoid arthritis were the six features that were identified as relevant predictors. Conclusions This prediction model would improve the overall patient care by enhancing prognosis counseling and informing treatment decisions for high-risk groups of actinomycotic osteomyelitis of the jaw. Supplementary Information The online version contains supplementary material available at 10.1186/s12903-022-02201-6.

Early diagnosis plays an important role in preventing the serious consequences of progressive osteomyelitis, such as pathologic fracture and deformity [1,7]. However, since AOJ is an infectious disease, it is difficult to diagnose based on clinical and radiological features. Microscopic examination and bacterial culture of the abscess are the gold standard method to diagnose AOJ [1,4,8]. However, administration of oral antibiotics before surgery leads to frequent false-negative results of the cultures in patients with osteomyelitis [6,9,10]. In addition, the surgical specimen used for the pathologic examination cannot be obtained until the necrotic bone is removed, thereby delaying the diagnosis. Thus, a new predictive approach using machine learning (ML) that can reflect the simultaneous analysis of various reported predisposing factors, including poor oral hygiene (such as dental caries, odontogenic infection), mucosal trauma (such as dental extraction), antiresorptive agent, gender, and diabetes mellitus, is required [8,11].
In recent years, an increasing amount of research applying ML techniques to medical classification has been conducted [12]. Their recent extensive application can be attributed to the increased availability of electronic health records [13]. However, there are very few published studies applying ML to osteomyelitis caused by an infection as direct identification or isolation of the infecting organism from a specimen of osteomyelitis may be laborious and time-consuming. Therefore, the purpose of this study was to develop and validate five ML models designed to predict AOJ to help provide guidelines for clinical decision-making and more effective treatment.

Methods
All experiments were performed in accordance with the guidelines and regulations approved by the Institutional Review Board (IRB No. 2020-06-002-0003) of Chungbuk National University Hospital and informed consent was obtained from all participants.

Study population and data collection
We retrospectively enrolled patients with osteomyelitis of the jaw treated in the Department of Oral and Maxillofacial Surgery, Chungbuk National University Hospital, South Korea, between January 2015 and June 2020. A representative case is shown in Fig. 1. Only patients who underwent sequestrectomy were included (Fig. 1a, b). The exclusion criteria were as follows: (1) multiple osteomyelitis of the jaw, (2) history of radiation therapy to the jaw, (3) patient loss during follow-up, and (4) incomplete medical records. The medical records of the patients were reviewed retrospectively to collect data, including age, gender, presumed causes, anatomical site, comorbidities, use of antiresorptive agents (ARA), use of antithrombotic agents, and recurrence. In total, 578 patient records were reviewed, and 222 patients were finally selected.

Histological analysis
The removed sequestrums were embedded in paraffin, cut into slices of 2 μm thickness, and stained using hematoxylin and eosin. A trained pathologist examined the slides for pathognomonic features of actinomycosis, such as sulfur granules. Photographs were taken of slides visualized by light microscopy. (Fig. 1c, d).

Machine learning
A schematic of the study design is shown in Additional file 1: Fig. S1 (see Additional file 1). Five ML methods, namely logistic regression (LR), random forest (RF), artificial neural network, support vector machine (SVM), and extreme gradient boosting (XGB) using the caret package provided in the R statistical software version 3.6.3 and R studio, (R Foundation for Statistical Computing, Vienna, Austria) were used to generate the prediction model [14][15][16]. The study design consisted of random splitting of the input dataset into training (n = 156; 70% of 222 patients) and testing (n = 66, 30% of 222 patients) datasets while maintaining equal proportions of the class ratios in each split. We developed five final ML models to predict actinomycotic infection in the training dataset by tuning the hyper-parameters using the caret package provided with the R statistical software (see Additional file 1: Table S1, Additional file 1, Additional file 2). We used five-fold cross-validation with 10 repeats to prevent overfitting. The Boruta algorithm based on random forest model was used to calculate the relative feature importance, which was provided in arbitrary units [17].

Statistical analysis
Statistical analysis was conducted using the R statistical software version 3.6.3 and R studio [14,15]. The frequency tables were analyzed using Student's t-test and the χ 2 test, as appropriate. The association between the variables and the AOJ-positive group was calculated using univariate regression analysis. The correlation between the two variables was demonstrated using Spearman's correlation analysis. P values < 0.05 (two-sided) were considered statistically significant. Five models were compared with each other and also with single predictors, such as age, using area under the receiver operating characteristic (ROC) curve (AUC) plotted using ggplot2 that is open-source data visualization package implemented in R [18]. ROC curves of single predictors in testing dataset including the age, gender, presumed causes, anatomical site, comorbidities, use of antiresorptive agents (ARA), use of antithrombotic agents, and recurrence were plotted. The AUCs were compared using the Delong test. The optimal threshold was calculated as the point closest to the top-left part of the plot. The performance metrics, including the accuracy, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) were obtained.

Results
The baseline characteristics of the patients are shown in Table 1. The age, proportion of females, the proportion of dental extraction and implants in the AOJ-positive group were significantly higher than that in the AOJ-negative group. Moreover, patients diagnosed with hypertension (HTN), cancer, patients using ARA, and recurrence were more common in the AOJ-positive group than in the AOJ-negative group. Interestingly, there was no recurrence in AOJ-negative group. In the correlation analysis, the AOJ-positive group highly correlated with three variables, namely patients using ARA (ρ = 0.53, p < 0.001), age (ρ = 0.37, p < 0.001), and presumed causes (ρ = − 0.41, p < 0.001) (Additional file 1: Fig. S2, see Additional file 1).
We performed a univariate regression analysis to identify the single independent feature associated with the AOJ-positive group (Fig. 2 Subsequently, we developed a prediction model using ML techniques. A schematic diagram of the prediction model development is shown in Additional file 1: Fig. S1 (see Additional file 1). The ratio of AOJpositive patients was 31.5% (70/222), which was consistent with the imbalanced data (Table 1). Therefore, we applied the oversampling methods to rebalance the training dataset. We subsequently tested all models using the testing dataset. The AUCs of all models were above 0.8, indicating that all models performed effectively in the testing dataset. The performance of ML, such as RF, SVM, and XGB, was significantly superior to that of the single predictor (such as age) (  Table S2; see Additional file 1). Lastly, the relative importance of all features was calculated using the Boruta algorithm [17]. Presumed causes, ARA, age, malignancy, rheumatoid arthritis, and HTN were the six features determined to be relevant in predicting AOJ-positive patients (Fig. 4). The performance of the prediction models, including accuracy, sensitivity, and specificity, PPV, and NPV is shown in Table 3.

Discussion
Herein, we developed ML-based models designed to predict the presence of Actinomyces in the jaw bone, which has not been previously attempted, to the best of our knowledge. We also included the performance metrics with the ROC curve and feature importance to enhance the interpretability of the ML models. All five prediction models exhibited comparable accuracy, and the value of the AUC (0.81 to 0.88) indicated excellent categorization regarding the predictive performance [19].
Multiple factors seem to affect the development of AOJ simultaneously. Therefore, clinicians often find it difficult to integrate these factors and their complex relationship with AOJ to guide treatment decisions-making. In our study, all ML models performed better than single predictors, namely age, suggesting that these models helped us analyze combinations of features to predict AOJ. It is noteworthy that combining only a few variables significantly increased the performance of the ML models, suggesting that a large number of variables is not essential to generate a good predictive model.
In recent years, ML approaches have gained popularity as a tool for all healthcare analysis, especially for medical image classification [20]. The greater availability of electronic medical records, as well as advances in hardware and software, have contributed to their recent widespread use. [21][22][23]. Despite these improvements in these approaches to classification tasks, current ML models, especially deep neural network, still operate like black boxes and fail to provide interpretations for their predictions [24]. It is also true that there are simple interpretable models such as LR. In the LR model, the coefficients helped us understand the cause of individual predictions. In our study, we used the Boruta algorithm based on the RF model to calculate the feature importance, which would allow clinicians to understand the relative importance of the variables involved in the overall prediction. Notably, presumed causes (such as extraction) were revealed as the most important risk factor in the relative feature importance calculated by the Boruta algorithm and regression analysis simultaneously. Since Actinomyces is a normal inhabitant of the oral cavity and lacks tissue-decomposing enzymes (such as hyaluronidases), mechanical trauma is the prerequisite that allows these endogenous microbial pathogens entry through the mucosal barrier and into the jaw leading to actinomycosis [4]. In line with this, our study revealed that the proportion of dental extractions and implants were significantly higher in the AOJ-positive group than that in the AOJnegative group.
In addition, ARA was revealed as the second risk factor following presumed causes in the relative feature importance calculated by the Boruta algorithm. Previous  studies have reported that Actinomyces species could be detected in about 80% of the samples from patients with medication-related osteonecrosis of the jaw (MRONJ) using histological techniques [10,11,25]. Consistent with this, our analysis showed that Actinomyces was present in 41 of 57 (71.9%) patients taking ARA. However, non-MRONJ patients showed a relatively low detection rate of Actinomyces (17.5%, 29 of 165) from the bone specimens, implying that Actinomyces was associated with the pathogenesis of MRONJ. Accurate causal inference and the role of Actinomyces underlying the development of AOJ remains elusive due to the lack of experimental validation in our study. It is still possible that actinomycosis is an opportunistic infection to pre-existent local osteomyelitis of the jaw bone. In the future, prospective studies investigating the microbiome originating from osteomyelitis of the jaw bone are needed to better understand the role of Actinomyces in the development of AOJ. Treatment standards for invasive actinomycosis have been developed and adapted from various studies and are based on prolonged antimicrobial treatment (such as amoxicillin with clavulanic acid) for 2-6 months combined with surgery [4,5,11,26]. Notably, recurrence was seen in only nine AOJ-positive patients in our study. Among those, six were administered antibiotics for less than 2 months, indicating the importance of extending antibiotic therapy. All patients with recurrence were completely cured after the prolonged administration of antibiotics and removal of the foci of infection, including sequestrectomy and excision of the granulation tissue until the sound bone was exposed. The capability for causal inference between recurrence and AOJ was limited due to the retrospective nature of this study.
A diagnosis of actinomycosis is best achieved by culture. However, the sensitivity of the culture is reduced significantly by the administration of antibiotics before sample collection [6,9,10]. In addition, special handling is needed to culture anaerobic organisms [27,28]. Therefore, histological examination is also preferred. Actinomyces species can be detected reliably in the affected bone specimens because of their morphologic appearance with staining [10]. In our analysis, Actinomyces species were histologically detected in 31.5% of the bone specimens, but there may still be an underestimation of the factual frequency of osteomyelitis associated with this infection. Recent developments in molecular methods such as 16 s rRNA target sequencing have revolutionized new approaches for the rapid detection of microorganisms, including those difficult to culture [29]. The sensitivity of the histological evaluation of clinical specimens for microbes is generally lower than microbiome analyses using sequencing technology since the latter involves an amplification step that increases the number of diagnostic targets. Thus, in the future, microbial detection using sequencing technology will likely be used for more accurate diagnosis.
This study has several limitations. The retrospective and cross-sectional nature of this study restricted causal inference. Further prospective studies should investigate the applicability of ML models for future prediction by transforming these retrospective data into a longitudinal research design. In addition, the overall performance of the ML techniques was comparable to that of LR. This result is mainly caused by the use of a dataset composed of categorical variables in this study. While the results are significant, we entitled this study as preliminary because of the limited number of patients and features. Furthermore, our analysis facilitated only speculation regarding the pathogenesis of AOJ with respect to various features owing to the lack of experimental validation in the ML technique. Recent advances in sequencing technologies and culture-independent methods have further elucidated the associations between the oral microbiome and oral health and disease state [29][30][31]. Additional studies using sequencing technologies are needed in the future to understand the microbial composition of the lesion in AOJ patients and achieve a rapid diagnosis.

Conclusions
Five prediction models exhibited comparable accuracy, and the range of the AUC results of 0.81 to 0.88 indicate good categorization in terms of predictive performance. The performance of the ML models, such as RF, SVM, and XGB was significantly superior to that of single predictors. Six features, such as presumed causes, antiresorptive agents, age, malignancy, hypertension, and rheumatoid arthritis were identified as relevant predictors. Hence, our prediction model, which considered various factors together as one complex, would improve the overall patient care by enhancing the prognosis counseling and informing treatment decisions to high-risk groups of AOJ.

Supplementary Information
The online version contains supplementary material available at https:// doi. org/ 10. 1186/ s12903-022-02201-6.   Fig. 4 Relative feature importance computed using the Boruta algorithm. Blue violin plots correspond to the minimal, average, and maximum Z scores of a shadow attribute. Red and green violin plots represent the Z scores of the rejected and confirmed attributes, respectively. Black dots and horizontal lines inside each violin plot represent the mean and median values, respectively. All features that received a lower relative feature importance than that of the shadow feature were defined as irrelevant for prediction